This invention relates to a system for creating facial images. It is applicable to the creation of composite pictures, specifically to systems used in law enforcement, artistic creations, recreation and education.
Human facial characteristics can convey a large amount of information and are a very common means for quickly identifying individuals. These facial characteristics are particularly useful in the area of law enforcement where composite facial images are often used to identify criminal suspects. They are also useful to create portraits in cases where no picture of the person to be graphically described is available.
Traditionally, these composite pictures are created by graphic artists who confer with an eyewitness. Through lengthy trial and error sessions, attempts are made to create a composite picture of a suspect.
The traditional method described above presents two problems. The first is that the accuracy of a composite picture is related to the graphic artist""s ability to translate the description of an eyewitness into an image. During the process, the memory of an eyewitness may be inadvertently altered by comments from the artist or from other third parties resulting in a less than accurate portrait of the suspect. The second problem with the traditional method is the high labor costs associated with the graphic artists themselves. Because police departments, in particular those in large urban centers, are likely to require several of such composite pictures, the associated labor costs are significant. Consequently, in order to overcome these problems, numerous attempts have been made to permit the witnesses to create the facial image with little or no interference from third parties. In addition to limiting the influence of a third party on the witness, the labor costs associated with the creation of composite pictures are also reduced.
One early technique for synthesizing single images of faces involved horizontally dividing the image of a face into bands for different features of the face such as hair, eyes, nose, mouth, and chin, respectively. Paper strips containing exemplary features could then be combined to form a composite drawing of a face.
One disadvantage of these systems is the presence of discontinuities between the various bands. For example, the size of the eyes may be disproportionate with that of the head or mouth. Furthermore, the skin tone between different bands may not be identical. These discontinuities often create and image which is cartoon-like and which bears little resemblance to the real life image the user of the system may wish to create.
Other techniques involve selecting individual components and combining them on a pre-selected face. In a typical interaction, the user first selects the shape of the face then eyes, nose, mouth and other components and combines them to form a facial image. Many variations on this theme can be used as described in Kakiyama et al. U.S. Pat. No. 5,600,767, Yoshino et al. U.S. Pat. No. 5,644,690, Sato et al. U.S. Pat. No. 5,537,662 and Belfer et al. U.S. Pat. No. 5,649,086 whose contents are hereby incorporated by reference. For example, the Sato et al. Patent, entitled Electronic Montage composing apparatus, describes a system for creating a montage image of a face using a plurality of basic parts stored in a library. In a typical interaction, the user selects a basic face from a plurality of basic faces and replaces the parts of the face in order to obtain the desired final result. Only the parts that are different from the ones desired need to be modified. This allows the user to quickly obtain a facial image without requiring the selection of each individual part. The Belfer et al. Patent describes another variation on this combination of individual components. This invention includes a database of exemplar images, or model components, which are stored and indexed according to various characteristics. Images are created by interpolating among the exemplar images in the database based on user selection of parameter values. Examples of high level parameters are gender and expression (surprise, anger). Examples of low level parameters are nose length, nostril width, hair thickness and so on. This system allows storing a reduced number of components by producing combinations of the components through interpolation. By allowing the user to vary parameters across a range of extreme values, facial images with the desired appearance can be created.
Although the use of individual components allows the user to position them at the desired location on the face, the proportions of the components often remains unsatisfactory. Furthermore, discontinuities remain and are often significant particularly when the facial images created comprise components selected from various genders and races. Furthermore in the case of the Belfer et al. Patent, tuning the parameters to obtain the desired appearance may be lengthy and difficult to master for the inexperienced user requiring the presence of an experienced technician.
Thus, there exists a need in the industry to refine the process of creating facial images such as to obtain a better composite picture system particularly applicable to law enforcement, artistic creations, recreational use and education.
An object of the invention is to provide a novel method and an apparatus for creating facial images.
Another object of this invention is to provide a computer readable storage medium containing a novel program element that directs a computer to perform the creation of a facial image.
Another object of this invention is to provide a computer readable storage medium containing novel library of basic morphological elements.
Another object of this invention is to provide a novel method for creating a library of basic morphological elements.
As embodied and broadly described herein the invention provides an apparatus for generating a composite facial image, said apparatus comprising:
machine-readable storage medium for storing a set of data elements, said data elements being representative of basic morphological elements, said data elements being in a format such that graphical representations of the basic morphological elements associated with the data elements are proportionate to one another;
an input for receiving user data indicative of a plurality of data elements;
processing means in operative relationship with said machine-readable storage medium for processing the plurality of data elements to provide a digital representation of a facial image resulting from a combination of the basic morphological elements associated with the plurality of data elements.
For the purpose of the specification, the expressions xe2x80x9cmorphological elementxe2x80x9d, xe2x80x9cmorphological componentxe2x80x9d and xe2x80x9cbasic morphological elementsxe2x80x9d are used to describe a part of a specific facial image. Examples of morphological elements are noses, eyes, mouths and eyebrows. In the preferred embodiment, morphological elements are grouped into morphological classes. For example, all nose elements are grouped into the xe2x80x9cNOSExe2x80x9d morphological class and all the eye elements are grouped in the xe2x80x9cEYExe2x80x9d morphological class. Optionally, the classes are divided into subclasses to further specify the types of elements stored. For example, the morphological class xe2x80x9cMOUTHxe2x80x9d may be subdivided into xe2x80x9cthick lipsxe2x80x9d, xe2x80x9cnarrowxe2x80x9d, xe2x80x9cthin lipsxe2x80x9d subclasses.
For the purpose of the specification, the expression xe2x80x9ccalibratingxe2x80x9d is used to designate the process of modifying the proportions of a set of distinct morphological components in order to improve their interaction. For example, the basic morphological elements in the library of basic morphological elements are calibrated to produce a facial image where the various parts can be connected with reduced discontinuities at their juncture. This is particularly useful when the volumes of the heads of different individuals from which the basic morphological elements were taken vary significantly.
For the purpose of the specification, the expressions xe2x80x9cskin texturexe2x80x9d and xe2x80x9cskin tonexe2x80x9d are used to designate the graphical appearance of the skin components of an image. The graphical appearance generally comprises the color shading of the skin. In the preferred embodiment a xe2x80x9cuniversalxe2x80x9d skin tone is used to facilitate the combination of different facial components by reducing the color contrasts of different skin types.
For the purpose of the specification, the expression xe2x80x9cnon-volatile storagexe2x80x9d is used to designate a machine-readable storage unit that maintains its contents even if the storage device has no power such as non-volatile RAM (NVRAM) or a hard disk.
In a most preferred embodiment of this invention, the composite picture system creates images of a face on the basis of images of individual facial parts, herein referred to as basic morphological elements. In the preferred embodiment, the composite picture system provides two modules namely a library of basic morphological elements and an image creation and modification module interacting with the library of basic morphological elements.
The image creation and modification module provides a means for creating a facial image. In a typical interaction, once the composite picture system is activated, the user of the system selects, through an interface, basic morphological elements from the library of basic morphological elements. These basic morphological elements represent images of facial parts such as eyes, noses, hair and so on. The operator interface to the composite picture system may be a keyboard, pointing device, touch screen or any other suitable command/data input means. Preferably for a given facial image, a single basic morphological element is selected from a given morphological class. Following this, the selected basic morphological elements are combined to form a facial image. In the preferred embodiment, voids between the part forming the mosaic of basic morphological elements are uniformly filled by a predetermined texture, herein referred to as the universal skin tone. The system then displays the facial image to allow the user to view it. The display may be a computer screen, a printed image or any other suitable means for displaying the facial image to the user. If the facial image is to the user""s satisfaction, the process is complete. If the facial image is not satisfying, the user may select other basic morphological elements through the user interface in order to get the desired result. Preferably, the composite picture system also includes vertical and horizontal controls allowing the user to modify the position of the basic morphological elements in the facial image. The user may modify the basic morphological elements until the image obtained is the one desired.
The second module of the composite picture system is the library of basic morphological elements. Each basic morphological element in the library is an image of a facial part or an accessory such as glasses, earrings or other. In a preferred embodiment, the basic morphological elements are collected by gathering photographs of individuals and isolating basic morphological elements. The accessories may be pictures or may be synthetically created with the use of graphing tools without detracting from the spirit of the invention. In a preferred embodiment, the system includes approximately 2,000 (two thousand) pictures of faces wherein selected individual basic morphological elements have been isolated. It is also desirable to include in the library of basic morphological elements images from both genders, images from a wide range of age groups and images from different races in order to have a good representation of the various facial characteristics of humans. Once the pictures have been collected, a processing operation is performed. The basic morphological elements are isolated and grouped according to the facial part they represent herein referred to as morphological class. In the preferred embodiment, the following basic morphological classes have been used: hairstyle, forehead, eyebrows, eyes, nose, mouth, chin, moustache, beard, wrinkles and glasses. Following this, a sequence of operations is performed on each individual basic morphological element in order to allow the facial parts to be compatible one with the other. As a first step the skin of the images are modified so all the skin parts of the basic morphological elements have a common skin tone herein referred to as universal skin tone. A filter operation may be applied to either darken the skin tone or lighten the skin tone. In the preferred embodiment, the facial image is created in a gray shade and the universal skin tone is a predetermined shade of gray. Preferably, the shade of gray is chosen such that it is neither dark nor pale such as to be able to represent a wide range of skin types. The selection of the shade of gray is subjective and depends on the preference of the designer and user. The basic morphological elements are then mapped onto a universal skin tone background. The following step involves a calibration operation of the basic morphological elements. This is of particular importance since the volume of the head of different individuals is variable. The variation in volume often creates discontinuities around the facial edges and the calibration step permits a reduction or elimination of these discontinuities. The calibration may also include positioning critical points in the facial image such as the eyes, nose, mouth and others at a predetermined position and interpolating the pixels in between the critical points. Once the calibration is complete, the basic morphological elements are cut into a shape suitable for use in the composite picture system. Optionally, once the processing operation has been completed, each basic morphological element is assigned a unique identifier, herein referred to as element code.
Typically, this identifier is a sequence of alphanumeric characters. A facial code for a facial image is created by combining the identifiers of the basic morphological elements constituting the facial image. The combination may be a concatenation of the element codes or any other means of combining the identifiers provided that the resulting facial code is unique to that set of basic morphological elements and their coordinates. The facial code allows each facial image to be described with a very small number of characters permitting the rapid transmission of the composite picture over data lines. For example, a police station A using the facial code can transmit the entire composite picture of a suspect to police station B by simply sending the code for that facial image. At the reception, police station B enters the code into the composite picture system that displays the composite picture. The communication between location A and location B may be a telephone line with a set of modems, an Ethernet connection or any other communication means suitable for the transfer of data.
In another preferred embodiment of the invention, the apparatus for creating facial images is integrated into a non-electronic composite picture system using a library of basic morphological elements in the form of physical modules on transparencies. The composite picture system provides two modules namely a library of basic morphological elements and an image combination fixture for holding the transparencies in a superposed relationship such that the morphological elements as the transparencies visually unite with one another to form a complete facial image.
The image combination fixture provides a handy tool for combining the basic morphological elements to create a facial image. In a typical interaction, the user selects a set of transparencies containing images of basic morphological elements from a library of basic morphological elements. These basic morphological elements represent images of facial parts such as eyes, noses, hair and so on. Following this, the selected basic morphological elements are combined to form a facial image. Typically the combination is done by superposing the selected transparencies in an order appropriate for the correct visioning of the image. The image combination fixture may be a simple tray designed to hold the transparencies in the proper alignment. As a variant, an overhead projector can be used with the transparencies to display the combined result on a screen. If the facial image is to the user""s satisfaction, the process is complete. If the facial image is not satisfying, the user may add a transparency or replace one of those selected previously until the desired result is obtained. Preferably, the vertical and horizontal coordinates of the basic morphological elements in the facial image can be changed by sliding the transparencies in the fixture or over the projector to the correct position.
In the preferred embodiment, the non-electronic composite picture system comprises a library of basic morphological elements. In the preferred embodiment, the basic morphological elements are collected by gathering photographs of individuals and isolating basic morphological elements. Once the basic morphological elements have been collected a processing operation is performed. In order to facilitate the processing operation, it is preferable to perform the processing operation with a general-purpose digital computer and to print the results on transparencies. The processing operation preferably includes the step of calibrating the basic morphological elements in order to allow the facial parts to be compatible one with the other and the step of modifying the skin tone of all the skin parts of the basic morphological elements such that the skin tone is uniform throughout the library. Optionally, once the processing operation has been completed, each basic morphological element is assigned a unique identifier. The coordinate part of the identifier is measure from the center point of the transparency and is the displacement with respect to a fixed point. Other methods for measuring the position of the transparencies may be used without detracting from the spirit of the invention. The facial code for a facial image may then be created by combining the identifiers.
As embodied and broadly described herein, the invention provides a method for generating a composite facial image, said method comprising the steps of:
providing a machine-readable storage medium for storing a set of data elements, said data elements being representative of basic morphological elements, said data elements being in a format such that graphical representations of the basic morphological elements associated with the data elements are proportionate to one another;
receiving user data indicative of a selection of a plurality of data elements;
processing the plurality of data elements to provide a digital representation of a facial image resulting from a combination of the basic morphological elements associated with the plurality of data elements.
As embodied and broadly described herein, the invention also provides a computer readable storage medium containing a program element for directing a computer to create a facial image, said computer including:
memory means including:
a set of data elements, said data elements being representative of basic morphological elements, said data elements being in a format such that graphical representations of the basic morphological elements associated with the data elements are proportionate to one another;
processor means in operative relationship with said memory means, said program element instructing said processor means for:
receiving user data indicative of a selection of a plurality of data elements;
processing the plurality of data elements to provide a digital representation of a facial image resulting from a combination of the basic morphological elements associated with the plurality of data elements.
As embodied and broadly described herein, the invention also provides a method for generating an electronic library of basic morphological elements suitable for usage in a composite picture system, said method comprising the steps of:
providing a plurality of facial images;
digitizing said facial images;
processing the digitized facial images to create a plurality of data elements, each data element being representative of a basic morphological element of a certain facial image;
calibrating said data elements such that all basic morphological elements are substantially proportional to one another;
storing the calibrated data elements in a computer readable storage medium.